@rip I had the very same issue and could not find any good solution posted already so I will post mine here (Z button was never an issue for me , Z button is also my hotkey)
Once configured as screenshot above and after first launch with n64 emu the following configuration was generated into /opt/retropie/configs/n64/InputAutoCfg.ini
; SWITCH CO.,LTD. Controller (Dinput)_START
[SWITCH CO.,LTD. Controller (Dinput)]
plugged = True
plugin = 2
mouse = False
AnalogDeadzone = 4096,4096
AnalogPeak = 32768,32768
Mempak switch =
Rumblepak switch =
C Button U = button(9)
L Trig = button(4)
C Button D = button(1) button(0) #C Down gets mapped to button(1) which is really B Button and button(0) is the actual C Down button
DPad U = hat(0 Up)
DPad L = hat(0 Left)
C Button L = button(3)
Y Axis = axis(1-,1+)
X Axis = axis(0-,0+)
DPad R = hat(0 Right)
Z Trig = button(6)
C Button R = button(8)
R Trig = button(5)
DPad D = hat(0 Down)
Start = button(12)
A Button = button(2) # Lack of mapping for button B at all
; SWITCH CO.,LTD. Controller (Dinput)_END
I then used jtest to actually view which button was what hence my conclusions above with jstest /dev/input/js0 then I edited the file nano /opt/retropie/configs/n64/InputAutoCfg.ini with the following
; SWITCH CO.,LTD. Controller (Dinput)_START
[SWITCH CO.,LTD. Controller (Dinput)]
plugged = True
plugin = 2
mouse = False
AnalogDeadzone = 4096,4096
AnalogPeak = 32768,32768
Mempak switch =
Rumblepak switch =
C Button U = button(9)
L Trig = button(4)
C Button D = button(0)
DPad U = hat(0 Up)
DPad L = hat(0 Left)
C Button L = button(3)
Y Axis = axis(1-,1+)
X Axis = axis(0-,0+)
DPad R = hat(0 Right)
Z Trig = button(6)
C Button R = button(8)
R Trig = button(5)
DPad D = hat(0 Down)
Start = button(12)
A Button = button(1)
B Button = button(2)
; SWITCH CO.,LTD. Controller (Dinput)_END
Hope this might help someone else with same setup, cheers
/a